Improving the Cross-Lingual Generalisation in Visual Question Answering

While several benefits were realized for multilingual vision-language pretrained models, recent benchmarks across various tasks and languages showed poor cross-lingual generalisation when multilingually pre-trained models are applied to non-English data, with a large gap between (supervised) English performance (zero-shot) transfer. In this work, we explore the of these on zero-shot visual question answering (VQA) task, where fine-tuned visual-question data evaluated 7 typologically diverse languages. We improve transfer three strategies: (1) introduce linguistic prior objective augment cross-entropy loss similarity-based guide model during training, (2) learn task-specific subnetwork that improves reduces variance without modification, (3) training examples using synthetic code-mixing promote alignment embeddings source target Our experiments xGQA multimodal transformers UC2 M3P demonstrates consistent effectiveness proposed fine-tuning strategy languages, outperforming existing methods sparse models.